Motion Picture Engineering
Video Streaming
1968
50 YEARS
70%
internet is
video
1 Billion
hours on
YouTube
40%
of the world
owns a
smartphone
And then the pandemic hit in 2020
•Real time video communications is one of the
technologies which allowed the world economy to survive
•PEOPLE WERE ABLE TO WORK AND STUDY FROM HOME
•Video Streaming sites enabled entertainment (SEE THIS)
•STREAMING VIEWERSHIP UP 30% AT PEAK OF 1ST LOCKDOWN IN AUG 2020
•The pressure on bandwidth was so large that
Governments appealed to Netflix to reduce their
streaming resolutions.
The challenges of video streaming
•The available bandwidth is generally not enough to
receive the video data generated at the source.
•That bandwidth changes every second.
•There are thousands of different realisations of
devices/hardware for generation and display of media
•Devices can generate and display different resolutions
and aspect ratios. They can be rotated in the middle of
playback so your aspect ratio can change suddenly.
•The artistic intent of a director in terms of picture quality
needs to be reproduced on every possible display device.
The challenges of video streaming
Challenge
Solution
The bandwidth is not enough
Video Compression/Transcoding/
Transrating
Bandwidth is time varying
Adaptive Bitrate (ABR) Compression,
HLS+DASH
1000’s of different devices
Adaptive Multiplexing and ABR
Reproducing Artistic Intent
Perceptual Quality Measurement
Back of the envelope calculation
Video Generated
Raw Data Bitrate
1080p from
iphone, at 25fps,
YUV420 sampling
Bits/Pixel
×Pixels/frame ×fps = 8 × 1920 ×
1080
× 1.5 × 25 = 593Mbps
4k from Red “Helium” Cinema
Camera at 60fps RGB444
sampling, 16 bit channel
=16 × 3840 × 2160 × 3 × 60 =22.8 Gbps
Your average phone bitrate is 8Mbps, Home Fibre probably 50Mbps
But that is only you. There are maybe 1B people looking at video in every instant
of every day. Each pixel has got to get to all of them. So the bandwidth available
is less.
Thankfully Video Compression Algorithms have improved and Quality has changed for the better
320 Kbps 160 Kbps
•Phone
Cameras
generate
20Mb/sec
•Movies from
Hollywood on
BluRay are
100’s Gbits/sec
•Billions of
uploaders
Mobile Goodput
is 5Mbits/sec
Copper 50
Mbits/sec
Cable 100
Mbits/sec
Battery life,
Compute power
finite and low
•Phone
Cameras
generate
20Mb/sec
•Movies from
Hollywood on
BluRay are
100’s Gbits/sec
•Billions of
uploaders
Mobile Goodput
is 5Mbits/sec
Copper 50
Mbits/sec
Cable 100
Mbits/sec
Battery life,
Compute power
finite and low
RATE/QUALITY TRADEOFF
Low bitrate but poor quality High quality but high bitrate
Video Compression Basics (very)
PP
I
P
PB B
I
DCT
Basic DSP Elements of Media Compression
A good site for leaning more
Reference Material
Background on Digital Media
http://xiph.org/video/vid1.shtml http://xiph.org/video/vid2.shtml
Excellent Overview of Media Compression at http://people.xiph.org/~tterribe/pubs/lca2012/auckland/intro_to_video1.pdf ;
https://www.xiph.org/daala/
VP9 Presentation at Google IO 2013 : http://www.youtube.com/watch?v=K6JshvblIcM
HEVC Information http://hevc.hhi.fraunhofer.de/
http://www.atlanta-smpte.org/HEVC-Tutorial.pdf
H.264 Information http://www.itu.int/rec/T-REC-H.264
Tools : www.ffmpeg.org http://www.videolan.org/
Rate Control in H.264 : http://www.pixeltools.com/rate_control_paper.html
Communicating Pictures
A Course in Image and Video Coding
1st Edition - June 20, 2014
2007
Vimeo starts
streaming HD
When did we start getting this right?
2000
2005
2010
2015
2010
Netflix
dominates
evening
streaming
2013
YouTube
launches
Denoiser, VP9
2014
4K
HFR
HDR
360 Video
2016
8K
3D 360
SORTING OUT INFRASTRUCTURE
2000
Sky Turns off
Analoge
2005
YouTube
H.264
H.264
H.263 H.265, VP9, Daala H.266, AV1
Theora, VC1, Dirac
MPEG2
6 Mb/sec, 720x576 6 Mb/sec, 1920 x 1080p
16 Mb/sec
TELEVISION BROADCAST LEADS OTT Leads Picture
Quality
Fukushima increased importance of ffmpeg and video over the internet
2000
2005
2010
2015
SKY
YouTube
ANALOG INGEST TO YOUTUBE, NETFLIX, iTUNES
Ingest
Fukushima
DIGITAL INGEST
4096 x 2160 (4K DC)
3840 x 2160 (4K UHD)
1920 x 1080 (1080p) Full HD
1280 x 720(720p) HD
640x360 (360p) SD
H.264@1Mb/sec
H.264@5Mb/sec
H.264@7Mb/sec
H.264@45Mb/sec
Some Trends (from YT): Resolutions
4k and 8k
Some Trends (from YT): Video Codecs
Interlaced media never goes
away
Some Trends: Colorspaces
SCALE DIVERSITY BANDWIDTH
A summary of the modern challenges
400 Hours
Uploaded
per Minute
Available
bandwidth
varies
enormously
Ingest Universal
Xcode
Xcode
Xcode
CDN
1080p
720p
480p
QoE Measurement
Thumbnails
Security
User Edit
Enhancements
Addressing scale, diversity and varying bandwidth
DASH
Dynamic Adaptive Streaming over HTTP
The Core Cloud Video Engine
Chunker
Parameters
Target Bitrate
Preprocess Transcode
Transcode
Transcode
Bitrate Too Large to Stream to You
Bitrate lower enough to stream to you
Parameters the
same for everything
Have to process in parallel
to keep up with ingest rate
TRANSCODERS ARE COMPLICATED
LOADS MORE PARAMETERS THAN ANYONE FIDDLES WITH
7 Mbits/sec 7 Mbits/sec
Both these clips are 1280x720 @ 30fps and are therefore
allocated the same amount of bits! Instead we should treat every
clip differently
SSIM = 0.78
SSIM = 0.99
SSIM = 0.53
PSNR = 28.3 dB
PSNR = 27.5 dB
PSNR = 27.5 dB
WOULD BE GREAT
IF WE COULD
WATCH
EVERYTHING
AND MEASURE
PICTURE QUALITY
: But we can’t. So we
develop algorithms
to measure quality.
With a quality measurement we can do this kind of thing : Adaptive Quality Control Pipeline
Inges
t
Universal
Xcode
Xcode
Xcode
Playout
1080p
720p
480p
Universal
Xcode++
Xcode++
Xcode++
Playout
1080p
720p
480p
Denoise
Measurement Popularity
Partner Identity : High
Value Content
*Media Modeling
*Defect Analysis
*Device Information
THIS IS WHAT WE ARE GOING TO DO
Challenge
Topic
Assessment
The bandwidth is
not enough
Review of Video
Compression Standards
Diversity
Transcoding
Lab on Transcoding (15%)
Diversity,
Bandwidth
Adaptive Bitrate (ABR)
Compression, ABR
Standards
Assignment on ABR (20%)
(Creating a bitrate ladder)
Reproducing
Artistic Intent
High Dynamic Range Video
and Modern Colour
Standards
AND Quiz worth 10%
FIN
